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1. Introduction 


Quantal response analysis is the methodology used to estimate the probability of a successful 
dichotomous response as a function of some stimulus variable. In a current research program 
being investigated at the U.S. Army Research Laboratory, a projectile is fired at a sample of 
developmental annor to determine if the target is able to defeat the round, i.e., if the round did 
not perforate the back side of the target. Projectile vs. target applications of quantal response 
analysis have traditionally utilized impact velocity as the stimulus variable. Realizing that other 
factors play a significant role in the interaction of a projectile and target, this current program 
considers a set of quantitative characteristics of the projectile measured from x-ray images prior 
to the moment of impact. These characteristics serve as input to a computer model that predicts 
how much penetration would have occurred if that same round was fired into semi-infinite rolled 
homogeneous annor (RHA). This predicted RHA penetration is an indicator of projectile 
lethality and therefore serves as the stimulus variable used in the quantal response analysis for 
this research program. 

Because quantal response analysis requires that the probability of success increases with an 
increase in the stimulus variable, one must be careful in defining a successful outcome. 

Although the purpose of the study is to develop an effective annor against this particular threat, a 
successful outcome occurs if the projectile completely penetrates the armor —not if the armor 
defeats the round. 


The underlying mathematical model of quantal response analysis assumes that each experimental 
unit has an associated threshold level of the stimulus variable for which an exceeded exposure 
will result in a successful response. In particular, each given annor sample has a corresponding 
RHA penetration threshold which is unknown and always will be unknown due to the destructive 
nature of the experiment. If the presented threat has an RHA penetration capability exceeding 
this threshold, then the armor will be penetrated; otherwise, the annor will not be penetrated. 


Additionally, the underlying model assumes that the population of threshold values is nonnally 
distributed. The objective of quantal response analysis is to estimate the parameters (mean and 
standard deviation) of this normal distribution. From a probabilistic perspective, the success 
probability associated with a given RHA penetration is the probability that a randomly selected 
annor’s threshold RHA penetration will be less than the given value. This probability is given 
by the cumulative distribution function (CDF) that results from these nonnal parameter 
estimates. That is, if r is a specific RHA penetration, then the probability of penetration into the 


developmental armor is given as P{r) = ® 


r- ju 


<j 


, where ju is the estimate of the mean, & is 


1 




the estimate of the standard deviation, and O ( ) is the standard normal CDF, a function that is 
tabulated in most elementary statistics texts. 

Estimates for the mean, //, and the standard deviation, cr, are obtained using maximum 
likelihood and the DiDonato-Jamagin algorithm. 1 For this algorithm to converge to a solution, 
the following conditions must be met: 

1. The set of responses must include both successes and failures. 

2. The average of the success-producing stimuli must exceed the average of the failure- 
producing stimuli. 

3. The maximum failure-producing stimulus must exceed the smallest success-producing 
stimulus. 

Figure 1 graphically shows what happens when the convergence conditions are not met, and 
when all of the conditions are satisfied in a quantal response study. For each case, a successful 
outcome is plotted as a “1” on the vertical axis; failures are plotted as “0.” The case in figure la 
occurs when all outcomes are successful, while the case in figure lb occurs when they are all 
failures; both of these cases are a violation of condition 1. Figure lc shows an example of when 
condition 2 is not met, i.e., the average of the success-producing stimuli is less than the average 
of the failure-producing stimuli. The case in figure Id is a violation of condition 3, whereby the 
minimum of the success-producing stimuli exceeds the maximum of the failure-producing 
simuli. An example for which all three convergent conditions are satisfied is shown in figure le. 

When condition 3 is met, the region between the smallest success-producing stimulus and the 
maximum failure-producing stimulus is referred to as the zone of mixed results (ZMR). If no 
ZMR exists, an alternate method is required to obtain estimates for the parameters // and <j . 

The method advocated in this report is an example of data augmentation, a statistical technique 
in which artificial and observed data are combined to facilitate the analysis. Spurred by the 
advent of high-speed computing, augmentation has gained favor among many in the statistical 
community and is the basis for several recently developed methods including Markov chain 
Monte Carlo, 2 the Expectation-Maximization Algorithm, 3 and problems involving missing data. 4 


1 DiDonato, A. R.; Jamagin, M. P. Maximum Likelihood Estimation in Quantal Response Experiments. SIAM J. Appl. Math 
1974 , 26 (2), 447-454. 

2 

“ Gamerman, D. Markov Chain Monte Carlo; Chapman & Hall: London, 1997. 

3 

Dempster, A.; Nan Laird, N.; Rubin, D. Maximum Likelihood from Incomplete Data via the EM Algorithm. J. of the Royal 
Statistical Society 1977 , 39 (1), 1-38. 

4 Kirk, R. E. Experimental Design ; 2nd ed.; Brooks/Cole: Monterey, CA, 1982. 
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Figure 1. Examples of the five possible cases in a quantal response study. 


2. Quantal Response Analysis When the ZMR Does Not Exist (Condition 3 
Failure) 


When a ZMR is absent, data augmentation as a method for estimating the mean and standard 
deviation of the threshold stimulus levels is proposed. This approach starts by generating a copy 
of the original data set and then adding a random noise component to each of the stimuli. The 
random noise components are independently drawn from a normal distribution with zero mean 
and a standard deviation equal to the precision with which the stimuli can be measured. This 
artificial data is combined with the original data to produce an augmented data set that is twice as 
large as the original data set and has the same ratio of successes to failures. If the augmented 
data contains a ZMR, then the quantal response analysis proceeds in the customary manner by 
estimating the normal parameters with the DiDonato-Jamagin algorithm. However, if there is 
still no ZMR, then the augmentation process is continued until a ZMR is obtained.* 


If the difference between the minimum success stimulus and the maximum failure stimulus (A) is large relative to the 
standard deviation (a*) used to generate the noise components, repeated augmentation may not necessarily yield a ZMR. As a 
general rule of thumb, it is recommended that a* > A/5 for augmentation to work efficiently. 
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A point estimate for either the probability of success at a given stimulus level or a particular 
percentile is easily calculated from the estimated // and cr. However, since this point estimate 

is based on one or more random artificial data sets, it will vary from analysis to analysis (i.e., 
separate analyses of the same original data will produce different solutions). This may not (and 
should not) be acceptable to most researchers. Therefore, it is recommended that confidence 
intervals (or bounds) be reported in lieu of point estimates. Confidence statements are made by 
repeating the augmentation analysis many times to produce an empirical distribution of point 
estimates. Quantiles from the empirical distribution form the appropriate confidence limits. 


3. Example of Augmented Quantal Response Analysis When the ZMR Does 
Not Exist 


Consider a quantal response study in which 10 projectiles are fired at samples of developmental 
armor to see if they result in a complete penetration. Based on the threat characteristics 
immediately prior to impact, a postshot computer analysis detennines how much penetration 
would have been achieved into semi-infinite RHA. These predicted RHA penetrations serve as 
the stimulus variable, while nonpenetration into the developmental armor serves as the 
dichotomous response. Eight of the threats failed to penetrate the developmental annor; their 
RHA predictions were 1026.1, 1027.2, 1035.4, 1051.8, 1052.7, 1060.0, 1073.8, and 1078.8.* 
Two of the threats successfully penetrated the armor with corresponding RHA predictions of 
1081.6 and 1084.3. Figure 2 shows these outcomes graphically. Note that the maximum RHA 
prediction from a nonpenetrating round is 1078.8 and the minimum RHA prediction is 1081.6. 
Therefore, a ZMR does not exist, and an analysis using augmented data is warranted. 



Figure 2. Data set from quantal response study with no 
zone of mixed results. 


Due to the sensitive nature of the actual study, the RHA penetrations reported here have been altered and the units are not 
specified. 
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The standard deviation used in generating all noise components for the artificial stimuli is 7, 
since this is the prediction error for the RHA penetration model. The first artificial value, drawn 
from a normal distribution having mean 1026.1, is a failure with RHA penetration 1024.8; the 
second artificial value, drawn from a normal distribution having mean 1027.2, is a failure with 
RHA penetration 1032.3; ...; the eighth artificial value, drawn from a normal distribution having 
mean 1078.8, is a failure with RHA penetration 1079.2. The ninth artificial value, drawn from a 
normal distribution having mean 1081.6, is a success with RHA penetration 1078.5; the tenth 
artificial value, drawn from a normal distribution having mean 1084.3, is a success with RHA 
penetration 1080.9. Combining the original data and the artificial data yields an augmented set 
of data, which is shown in figure 3. Now the maximum failure occurs at 1081.3 and the 
minimum success at 1078.5, so that a ZMR exists. 



Figure 3. Augmented data set from quantal response 
study with no zone of mixed results. 

The DiDonato-Jamagin algorithm returns estimates for the mean and standard deviation of 
1079.96 and 2.78, respectively. The normal cumulative distribution function for these parameter 
values (see figure 4) displays the relationship between any RHA penetration prediction and the 
estimated probability in penetration into the developmental armor. For example, figure 4 also 
shows that at 1083 units of RHA predicted penetration, the probability of penetration through 
developmental armor is estimated to be 86.3%. 



Figure 4. Quantal response curve from augmented data set. 
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If one repeats this analysis often enough, the resulting empirical distribution can be used to 
generate a confidence interval for the penetration probability at an RHA penetration of 1083 
units. Figure 5 gives a sense of the distribution of these point estimates based on 10,000 
repetitions of the analytical process. The point estimates span a considerable range of values. A 
90% confidence interval is obtained by choosing the 500th smallest point estimate as the lower 
limit and the 500th largest point estimate as the upper limit.* The resulting 90% confidence 
interval is 49.0% and 98.9%. This interval is quite wide, most likely due to the small number of 
shots (10) that the analysis is based upon. 



Estimated P(penetration) for RHA penetration of 1083 


Figure 5. Frequency histogram for 10,000 estimates of 

developmental armor penetration probability for a 
predicted RJfA penetration of 1083 units. 


4. Quantal Response Analysis When the Responses Are Identical (Condition 
1 Failure) 


Suppose that all of the experiments in a quantal response study result in identical outcomes. Due 
to limitations on test resources, it may still be desirable to estimate the probability of success at a 
particular stimulus level. Assume that a study yields successful outcomes at each of the stimulus 
levels considered. A conservative approach for producing a quantal response curve involves 
changing the minimum success-producing stimulus level to a failure and then using the 
augmentation routine detailed in section 2 of this report. * 1 ' This is likely to induce an upward bias 
in the estimated mean of the threshold stimuli, meaning that ju is, in general, larger than the true 
value ju . In turn, this approach will also tend to underestimate success probabilities. For this 


In general, a 100-(l-a) percentile two-sided confidence interval is bounded below by the lOO-a/2 percentile and above by the 
100-(l-a)/2 percentile of the distribution of point estimates. 

1 1f all of the observed outcomes are failures, change the maximum stimulus to a success and proceed with the augmentation 
routine. 
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reason, it is advised that this method only be used to produce lower confidence bounds for 
success probabilities at stimulus levels that are greater than (or very close to) the minimum 
success-producing stimulus level. 


5. Example of Augmented Quantal Response Analysis When All Responses 
Are Successes 


Suppose that a quantal response study similar to that described in section 3 yields all penetrations 
at the following predicted RHA penetrations: 1026.1, 1027.2, 1035.4, 1051.8, 1052.7, 1060.0, 
1073.8, 1078.8, 1081.6, and 1084.3. The data is changed so that a failure is assumed to have 
been observed at the RHA penetration of 1026.1. An analysis is then outlined in sections 2 and 
3. Figure 6 shows the alteration to the data. 



Figure 6. Example of quantal response data in which all 
outcomes are successes. The outcome with the 
smallest stimulus level is changed to a failure to 
facilitate the analysis. 

Estimating the probability of penetration for RHA predictions of 1028 and 1080 is required. A 
large sample of point estimates is then generated for each of these quantities using data 
augmentation. Figure 7 shows the resulting frequency histograms, giving a sense of the range of 
possible estimates. 

Lower confidence bounds for the developmental armor penetration probabilities at RHA 
predictions of 1028 and 1080 are 49.5% and 99.97%, respectively. It is clearly seen that there is 
far greater uncertainty in estimating the success probability at the smaller stimulus level of 
1028 units. This is because there is relatively little information about outcomes for stimuli below 
1028 units compared to the amount of information kn own about outcomes below 1080 units. 
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Estimated P(penetration) for 
RHA penetration of 1028 



Estimated P(penetration) for 
RHA penetration of 1080 


Figure 7. Frequency histograms for 10,000 estimates of developmental armor penetration probability for predicted 
RJfA penetrations of (a) 1028 units and (b) 1080 units. 


6. Quantal Response Analysis When the Average Success-Producing Stimulus 
Is Less Than the Average Failure-Producing Stimulus (Condition 1 
Failure) 


If the average of the success-producing stimuli is less than the average failure-producing stimuli, 
then the tendency is for the relationship between stimulus level and probability of success to be 
decreasing. This contradicts the assumption that this relationship should be increasing. To get 
around this problem, simply reverse the definitions of success and failure in the study. Under the 
new definition, proceed with the customary quantal response analysis or the augmentation 
approach outlined heretofore. Of course, it is necessary to convert the results back into the 
language and values of the original definition of a successful outcome. 

Because this is a trivial case, no example is given. However, it is worth reiterating that the 
RHA-prediction examples described in sections 3 and 5 were motivated by a survivability- 
oriented study in which success was originally defined to occur if the armor defeated the 
projectile. 


7. Summary 


In this report, the statistical concept of data augmentation has been applied to quantal response 
analysis when the conditions guaranteeing convergence of the DiDonato-Jamagin algorithm are 
not satisfied. This approach was primarily motivated by a need to perform the analysis when a 
zone of mixed results did not exist. Combining the original data with one or more artificial data 
sets that have been altered by a random noise component added to the stimulus levels has been 
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postulated. The augmented data set does have a zone of mixed results, allowing one to proceed 
with the usual maximum-likelihood method using the DiDonato-Jamagin algorithm. 

This novel approach leads to many questions and opportunities for future research. Topics that 
should be explored include unbiasedness of the estimates, the effects of sample size and choice 
of standard deviation, and other strategies for generating the artificial data. 
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